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EXECUTIVE  SUMMARY 


Problem 

In  recent  years,  medical  providers  and  the  scientific  community  have  become  increasingly 
aware  that  iatrogenic  illness  and  injury  are  a  major  source  of  morbidity  and  mortality  for 
hospitalized  patients.  However,  little  is  known  about  the  incidence,  nature,  and  causes  of 
adverse  events  in  medical  settings.  Systematic  knowledge  of  these  events  is  the  first  step 
toward  preventing  them. 

Objective 

The  objective  of  this  project  was  to  develop  a  survey  that  could  be  used  to  assess  adverse 
medical  events  in  an  Intensive  Care  Unit  (ICU).  The  survey  was  designed  to  gather  data 
about  adverse  medical  events  involving  hospitalized  patients  so  that  factors  associated 
with  these  events  and  possible  root  causes  could  be  identified. 

Approach 

A  method  called  the  critical  incident  technique  was  used  to  design  a  computer- 
administered  survey,  the  Computerized  Critical  Incident  Technique  (CCIT)  survey.  The 
survey  was  designed  to  obtain  information  from  medical  providers  on  adverse  medical 
events  and  near  misses  in  the  ICU.  A  test  version  of  the  CCIT  survey  (CCIT-I)  was 
installed  in  the  ICU  of  a  major  military  hospital.  The  survey  was  used  for  a  26-month 
period. 

Results 


Data  collection  with  the  CCIT-I  survey  generated  482  incident  reports.  The  survey 
system  was  positively  received  by  the  staff  and  the  reports  resulted  in  a  number  of 
corrective  actions  being  taken.  However,  examination  of  the  survey  and  of  the  data 
collected  with  the  survey  revealed  several  problems.  These  problems  were  due  primarily 
to  the  survey’s  heavy  reliance  on  closed-ended  questions  and  to  the  survey’s  branching 
strucmre,  which  precluded  comparability  of  data  across  respondents.  Because  of  these 
problems,  a  major  revision  of  the  survey  was  performed. 

The  second  version  of  the  survey  (CCIT-II)  was  evaluated  by  eight  subject  matter  experts 
to  determine  users’  reactions  and  to  detect  problems  with  the  survey.  In  general,  the 
subject  matter  experts  had  mostly  positive  cormnents  and  feedback.  They  provided 
suggested  changes  which  were  incorporated  into  the  next  revision  of  the  survey.  This 
resulted  in  the  third,  current  version  of  the  survey  (CCIT-in),  which  is  presented  in  the 
Appendix. 
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Conclusions 


The  potential  for  obtaining  valuable  information  about  adverse  medical  events  using  the 
CCIT-ni  survey  appears  promising.  We  recommend  that  the  CCIT-III  survey  or  a  similar 
survey  be  used  in  health  care  settings  so  that  recurring  iatrogenic  incidents  can  be 
identified  and  steps  can  be  taken  to  prevent  them. 


ABSTRACT 


Iatrogenic  illness  and  injury  are  a  major  source  of  morbidity  and  mortality  for 
hospitalized  patients.  However,  little  is  known  about  the  incidence,  nature,  and  causes  of 
adverse  medical  events.  The  objective  of  this  project  was  to  develop  a  survey  that  could 
be  used  to  assess  adverse  medical  events  in  an  Intensive  Care  Unit  (ICU).  The  survey 
was  designed  to  obtain  systematic  information  about  adverse  medical  events  so  that 
factors  associated  with  these  events  and  root  causes  might  be  identified.  The  critical 
incident  technique  was  used  to  develop  the  Computerized  Critical  Incident  Technique 
(CCIT)  survey.  A  test  version  of  the  survey  (CCIT-I)  was  installed  in  the  ICU  of  a  major 
military  hospital.  Data  collection  with  the  survey  generated  482  incident  reports.  The 
survey  system  was  positively  received  by  the  staff  and  the  reports  resulted  in  a  number  of 
corrective  actions.  A  revision  of  the  survey  was  undertaken  to  correct  several  problem. 
The  second  version  of  the  survey  (CCIT-II)  was  evaluated  by  eight  subject  matter  experts. 
The  comments  and  suggestions  provided  by  the  subject  matter  experts  were  used  as  the 
basis  for  a  second  revision  of  the  survey.  This  resulted  in  the  third,  current  version  of  the 
survey  (CCIT-ni).  The  potential  for  obtaining  valuable  information  about  adverse 
medical  events  using  a  survey  tool  like  the  CCIT  survey  appears  promising. 
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INTRODUCTION 


Human  error  occurs  in  all  aspects  of  life,  including  medical  settings.  In  medical 
settings,  however,  the  consequences  of  error  can  be  particularly  serious,  resulting  in  the 
injury  or  death  of  patients.  In  recent  years,  medical  providers  and  the  scientific 
community  have  become  increasingly  aware  that  iatrogenic  injury  and  illness  are  a  major 
source  of  morbidity  and  mortality  for  hospitalized  patients. 

Adverse  Events  in  Medical  Settings 

latrogenics  literally  means  “doctor-caused.”  Iatrogenic  injuries  and  illnesses  are 
adverse  effects  related  to  the  rendering  of  medical  care,  which  are  not  a  direct  or  indirect 
complication  of  the  patient’s  primary  condition  or  disease  (Perper,  1994,  p.  28).  A 
number  of  somewhat  confusing  terms  are  currently  used  in  the  literature  to  designate 
iatrogenic  events.  These  terms  include  adverse  events,  medical  mishaps,  critical 
incidents,  medical  accidents,  care-related  complications,  therapeutic  misadventures, 
iatrogenic  misadventures,  and  nosocomial  diseases  (Perper,  1994,  p.  28). 

Little  is  known  about  the  incidence,  nature,  and  causes  of  adverse  events  in 
medical  settings.  Despite  the  importance  of  this  topic,  only  a  relatively  small  number  of 
studies  have  examined  adverse  medical  events  in  a  systematic  way  (Abramson,  Wald, 
Grenvik,  Robinson,  &  Snyder,  1980;  Bigby  et  al.,  1987;  Dubois  &  Brook,  1988;  Gopher 
et  al.,  1989;  Leape  et  al.,  1991;  Wright,  Mackenzie,  Buchan,  Caims,  &  Price,  1991).  As 
recently  as  1994,  Van  Cott  (1994)  stated  that  good  data  on  the  nature  and  extent  of  errors 
in  clinical  care  do  not  exist.  The  most  extensive  and  well  known  of  the  published  studies 
is  the  Harvard  Medical  Practice  Study.  This  study  found  that  almost  4%  of  all  patients 
hospitalized  in  New  York  state  suffered  an  adverse  event  resulting  in  prolongation  of 
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hospital  stay  or  disability  at  time  of  discharge  (Brennan  et  al.,  1991;  Dubois  &  Brook, 
1988;  Leape  et  al.,  1991).  Of  these  adverse  events,  more  than  two-thirds  were  considered 
preventable.  Based  on  this  study,  it  has  been  estimated  that  over  a  million  patients  are 
injured  annually  in  hospitals  in  the  U.  S.  by  medical  treatments  intended  to  help  them 
(Leape,  1994).  Adverse  medical  events  actually  account  for  more  deaths  than  all  other 
types  of  accidents  combined  (Institute  of  Medicine,  1985). 

Adverse  events  are  particularly  likely  to  occur  in  the  more  dynamic  medical 
domains  such  as  emergency  rooms,  operating  rooms  and  intensive  care  units,  which  deal 
with  complex,  high-risk,  acute  patient  care.  The  combination  of  intense  and  conflicting 
time  demands,  the  dynamic  nature  of  the  environment,  and  the  seriousness  of  the  medical 
conditions  encountered  in  these  units  increases  the  risk  for  iatrogenic  events  (Bogner, 
1994a;  Gaba,  1994). 

Categories  of  Adverse  Events 

Adverse  events  that  occur  in  medical  settings  have  been  categorized  in  many 
different  ways  in  the  research  literature.  Typically,  adverse  medical  events  are  discussed 
in  terms  of  what  was  done  wrong  or  the  type  of  error  that  was  made  (e.g.,  wrong 
medication,  wrong  diagnosis).  Some  of  the  more  common  categories  of  adverse  events 
include  those  related  to  diagnosis,  communication,  equipment  failure,  medication  errors, 
and  procedures.  Diagnosis-related  events  include  errors  in  diagnoses,  failure  to  employ 
the  appropriate  diagnostic  tests,  and  failure  to  act  appropriately  based  on  the  test  results. 
Communication-related  events  include  failure  of  staff  in  charge  of  a  patient  to  convey 
crucial  information  to  each  other,  failure  to  obtain  information  from  a  patient  or  from  a 
patient’s  chart,  and  misinterpretation  of  written  orders  (e.g.,  due  to  illegible  handwriting). 
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Equipment  failure  includes  true  equipment  malfunction  as  well  as  user  failure,  which  may 
be  caused  by  a  wide  range  of  factors,  such  as  unfamiliarity  with  the  equipment  and  poor 
equipment  design.  Medication  errors  include  failure  to  administer  an  ordered  medication, 
administering  the  wrong  medication,  and  administering  the  wrong  dose  of  a  medication. 
Problems  related  to  procedures  include  use  of  inappropriate  procedures  and  substandard 
performance  of  procedures. 

Current  Perspective  on  Human  Error 

In  the  past,  efforts  to  reduce  mishaps  in  medical  environments  and  other  settings 
(e.g.,  aviation,  nuclear  plants)  have  focused  on  identifying  the  individual  or  individuals 
who  made  the  error  or  caused  the  mishap.  However,  this  has  not  been  an  effective 
method  for  improving  quality  of  care  (Bogner,  1994b).  Recently,  the  trend  in  human 
error  research  has  been  a  more  general,  systems  approach  to  the  study  of  mishaps 
(Bogner,  1994a,  1994b;  Gaba,  1994).  In  this  approach,  errors  are  viewed  as  evidence  of 
systems  failures,  rather  than  the  fault  of  individuals.  Analysis  of  errors  or  mishaps  is 
intended  to  uncover  “root  causes”,  which  are  often  embedded  in  the  overall  system  or 
organization.  As  pointed  out  by  Leape  (1994),  human  error  may  seem  to  be  the  cause  of  a 
patient  inadvertently  getting  one  drug  instead  of  another,  but  the  fact  that  the  labels  or 
containers  are  nearly  identical  is  a  systems  fault  that  is  easy  to  correct.  Most  experts  in 
human  error  (e.g.,  Bogner,  1994a;  Cook  &  Woods,  1994)  now  believe  that  a 
comprehensive  approach  to  the  assessment  of  mishaps  requires  an  examination  of  the 
broader  organizational  context  in  which  mishaps  occur. 

Research  on  adverse  medical  events  has  taken  a  variety  of  approaches,  including 
patient  chart  reviews  (Dubois  &  Brook,  1988;  Leape  et  al.,  1991),  the  use  of  incident 
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report  forms  (Abramson  et  al.,  1980;  Brennan  et  al.,  1991;  Gopher  et  al.,  1989), 
questionnaires  (Wright  et  al.,  1991),  and  interviews  (Bigby  et  al.,  1987;  Cooper, 
Newbower  &  Kitz,  1984).  Each  approach  has  various  strengths  and  weaknesses 
associated  with  it. 

Critical  Incident  Approach 

One  approach  that  is  useful  for  studying  adverse  medical  events  is  Flanagan’s 
(1954)  critical  incident  technique.  The  critical  incident  technique  is  a  flexible  set  of 
procedures  for  gathering  information  about  behavior  in  specific,  real-life  situations 
(Flanagan,  1954).  First  developed  in  the  second  World  War  by  Flanagan,  it  is  a 
systematic,  open-ended  procedure  for  eliciting  information  from  respondents  about 
specific  incidents  that  they  observed.  The  critical  incident  technique  has  been  used  in 
hundreds  of  studies  in  a  wide  range  of  domains,  ranging  from  education  to  medicine. 

One  of  the  main  strengths  of  the  technique  is  that  it  capitalizes  on  people’s  ability  and 
willingness  to  tell  stories  or  anecdotes  about  their  experiences,  yet  it  allows  this 
information  to  be  gathered  in  a  structured,  systematic  way.  Cooper  et  al.  (1994)  used  the 
critical  incident  technique  to  study  medical  mishaps;  other  researchers  (Bradley,  1992a; 
1992b;  Norman,  Redfem,  Tomalin,  &  Oliver,  1992)  have  used  this  technique  for  studying 
other  aspects  of  patient  care. 

Most  critical  incident  technique  researchers  (e.g.,  Flanagan,  1954;  Norman  et  al., 
1992)  view  a  critical  incident  as  a  complete  and  clearly  demarcated  scene  observed  first¬ 
hand  by  a  respondent.  Flanagan  defined  an  incident  as  “any  observable  human  activity 
that  is  sufficiently  complete  in  itself  to  permit  inferences  and  predictions  to  be  madp: 
about  the  person  performing  the  act”  (Flanagan,  1954,  p.  326).  Incidents  are  deemed 
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critical  when  the  purpose  of  the  action  and  the  outcome  of  the  incident  are  reasonably 
clear  to  the  respondent  or  observer  and  are  clearly  relevant  to  the  phenomenon  under 
study  (Flanagan,  1954).  (“Critical”  refers  to  an  incident’s  relevance  to  the  outcome  and 
not  to  the  severity  of  its  effect.)  For  an  incident  to  be  usable  for  research  purposes,  the 
respondent  or  observer  must  have  a  detailed  knowledge  of  both  the  incident  and  its 
context. 

In  the  critical  incident  technique,  critical  incidents  are  obtained  through  interviews 
or  questionnaires  with  individuals  who  are  active  in  the  domain  under  consideration  (e.g., 
medicine  or  aviation).  Most  of  the  questions  are  open-ended,  an  important  feature  of  the 
technique  because  it  yields  descriptions  of  incidents  and  their  possible  causes  that  are 
richer,  more  detailed,  and  more  comprehensive  than  information  obtained  from  closed- 
ended  questions.  After  incidents  have  been  collected,  they  can  be  examined  post-hoc  by 
judges  who  categorize  the  incidents  and  try  to  identify  causes  (Bradley,  1992b;  Flanagan, 
1954). 

An  important  benefit  of  the  critical  incident  technique  is  that  it  allows  near  misses 
to  be  studied,  along  with  actual  adverse  events.  Inclusion  of  near  misses  is  important 
because  it  allows  data  on  a  larger  set  of  incidents  and  causal  variables  to  be  obtained. 
Research  on  errors  in  aviation  suggests  that  the  study  of  near  misses  can  give  valuable 
clues  to  systems  problems  and  other  factors  that  might  eventually  lead  to  accidents. 
Quality  Assurance 

Both  the  Department  of  Defense  and  the  Joint  Commission  on  Accreditation  of 
Healthcare  Organizations  (JCAHO,  1996)  require  that  hospitals  and  ambulatory  care 
facilities  have  quality  assurance  programs.  The  JCAHO  (1996)  sets  forth  performance 
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improvement  standards  that  hospitals  and  other  health  care  facilities  should  strive  to 
achieve.  For  example,  hospitals  are  required  to  establish  guidelines  for  quality  assurance 
and  risk  management  and  to  maintain  records  that  provide  information  on  the  quality  of 
patient  care.  Although  hospitals  have  to  collect  information  internally  on  adverse  events, 
they  are  not  required  by  the  JCAHO  to  report  adverse  medical  events  in  any  standardized 
way  or  to  maintain  standardized  data  bases  on  adverse  events.  Most  hospitals  use  one  or 
more  of  the  following  techniques  to  gather  quality  assurance  data:  incident  reports, 
anonymous  questionnaires,  case  reviews,  patient  chart  reviews,  administrative  data,  and 
observation.  There  are  a  host  of  problems  with  these  methods,  including  biased 
reporting,  incompleteness  of  information,  failure  to  report  all  adverse  incidents,  and  the 
lack  of  comparability  of  measurement  across  patients  and  departments.  Clearly,  health 
care  facilities  need  more  standardized,  objective,  and  comprehensive  methods  for 
assessing  adverse  events  and  identifying  their  causes. 

Objective 

The  objective  of  the  present  project  was  to  develop  a  survey  for  assessing  adverse 
medical  events  using  the  critical  incident  technique.  The  purpose  of  the  survey  was  to 
allow  the  collection  of  information  from  hospital  personnel  on  critical  incidents  (adverse 
events  and  near  misses)  occurring  in  an  Intensive  Care  Unit  (ICU).  The  ultimate  goal  was 
to  use  the  data  to  make  changes  to  reduce  adverse  events  and  improve  patient  care. 

DEVELOPMENT  OF  THE  CCIT  SURVEY 

An  adaptation  of  the  critical  incident  technique  was  used  to  design  a  computer- 
administered  questionnaire,  the  CCIT  (Computerized  Critical  Incident  Technique)  survey, 
for  collecting  information  about  adverse  medical  events.  The  survey  also  collected 
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information  about  other  aspects  of  the  medical  work  environment  that  might  have  an 
impact  on  patient  care. 

The  first  version  of  the  CCIT  survey  (CCIT-I),  which  is  described  below,  was 
developed  and  tested  by  Kobus,  Amundson,  Gubler,  Rascona,  Pang,  and  Van  Orden 
(1996).  Additional  information  about  this  version  of  the  survey  is  provided  in  Kobus  et 
al.  (1996).  The  subsequent  versions  of  the  survey  were  developed  by  the  present  authors 
(Booth-Kewley  and  Freeman). 

Description  of  the  CCIT-I  Survey 

The  CCIT-I  survey  elicited  information  about:  (1)  adverse  events  -  incidents 
involving  patient  discomfort  or  morbidity  outside  the  primary  disease  process,  (2)  near- 
misses  -  incidents  that  did  not  lead  to  an  adverse  event,  (3)  positive  events  -  nonstandard 
actions  by  health  providers  that  had  a  positive  impact  on  the  patient,  and  (4)  workplace 
concerns  -  any  issue  or  concern  regarding  the  workplace  environment. 

The  “core”  of  the  survey  consisted  of  questions  asking  respondents  to  describe 
actual  medical  mishaps  (adverse  events  and  near  misses)  they  had  participated  in  or 
observed  recently  in  the  ICU.  After  describing  an  incident,  the  respondent  was  asked  to 
categorize  the  incident  according  to  what  the  respondent  believed  to  be  the  primary  causal 
factor  (e.g.,  equipment,  communications,  procedures).  The  follow-on  questions  differed 
at  this  point  depending  upon  the  category  selected  by  the  respondent.  For  example,  a 
respondent  who  thought  his/her  incident  belonged  in  the  equipment  category  was  asked 
detailed  questions  about  the  type  of  equipment  involved,  whereas  a  respondent  who 
classified  an  incident  as  involving  procedures  was  asked  questions  related  to  procedure 
type  and  performance.  Questions  were  also  asked  about  primary  and  secondary  causes  of 
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the  incident,  the  immediate  and  secondary  consequences  to  the  patient,  whether  the 
incident  was  preventable,  and  what  could  be  done  to  avoid  such  incidents  in  the  future. 
This  information  was  collected  so  that  problems  could  be  identified  and  quantified, 
possible  causal  factors  could  be  identified,  and  interventions  designed. 

The  first  version  of  the  CCIT  survey  (CCIT-I)  was  constructed  in  a  computer- 
administered  mode  to  make  it  easier  and  more  interesting  for  respondents  to  complete  and 
eliminate  the  need  for  data  entry,  thereby  expediting  feedback  to  the  providers.  The 
computerized  mode  of  administration  prevents  missing  data  and  branching  errors  which 
frequently  occur  on  paper  surveys  (Rosenfeld,  Booth-Kewley,  &  Edwards,  1993).  The 
survey  was  administered  on  a  Macintosh  computer,  in  an  easy-to-use  format. 

Respondents  were  prompted  with  complete  instructions  on  each  screen  and  did  not  have 
to  have  computer  experience  to  use  the  system. 

Data  Collection  with  the  CCIT-I  Survey 

The  CCIT-I  survey  was  installed  in  the  ICU  at  San  Diego’s  Naval  Medical  Center 
(Balboa  Hospital).  It  was  used  for  a  26-month  period.  Staff  members  (e.g.,  nurses, 
physicians,  corpsmen)  were  encouraged  to  complete  surveys  for  any  incident  they  had 
participated  in  or  observed.  Surveys  were  completed  anonymously;  no  identifying 
information  was  obtained  from  respondents. 

Using  the  CCIT-I  survey,  482  cases  were  reported  by  medical  staff  at  the  San 
Diego  Naval  Medical  Center’s  ICU.  These  cases  included  149  adverse  events,  157  near 
miss  incidents,  157  workplace  concerns,  and  19  positive  events.  Since  the  survey  was 
anonymous,  and  because  some  respondents  completed  multiple  surveys,  we  cannot 
determine  the  total  number  of  individuals  who  responded.  The  survey  system  was 
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positively  received  by  the  staff,  and  the  reports  resulted  in  a  number  of  corrective  actions 
taken  to  reduce  the  likelihood  of  future  adverse  events. 

Problems  with  the  CCIT-I  Survey 

An  examination  of  the  first  version  of  the  CCIT  survey  (CCIT-I)  and  the  data 
identified  several  problems.  A  major  defect  was  the  fact  that  the  survey  forced  the 
respondent  to  classify  each  adverse  event  or  near  miss  incident  into  a  single  discrete 
category  early  on  in  the  survey.  As  a  result,  many  instances  of  seemingly  similar 
incidents  were  categorized  differently  by  different  respondents.  For  example,  a  fairly 
large  portion  of  the  adverse  events  and  near  misses  involved  problems  with  the  insertion 
of  a  line  or  tube.  This  problem  was  classified  by  some  users  as  a  "procedure"  problem,  by 
some  as  a  "medication"  problem,  and  by  others  as  an  "equipment"  problem.  Examination 
of  the  narrative  information  failed  to  reveal  systematic  reasons  for  these  differences  in 
categorization.  It  appeared  that,  in  the  absence  of  stated  criteria  for  categorizing  events, 
respondents  classified  events  arbitrarily  or  according  to  their  own  subjective  criteria. 

The  classification  inconsistencies  noted  above  produced  an  additional  problem. 
The  CCIT-I  survey  was  designed  so  that  the  initial  problem  category  chosen  by  the  user 
determined  (via  branching)  the  set  of  questions  the  user  was  subsequently  asked.  As  a 
result,  different  sets  of  follow-on  questions  were  presented  to  respondents,  depending 
upon  their  initial  choice  of  category  (e.g.,  procedure,  equipment).  This  meant  that  the 
data  collected  from  one  respondent  could  be  very  different  from  the  data  collected  from 
another  respondent  even  when  the  incidents  (e.g.,  intubation  problems)  were  similar. 
Consequently,  a  standardized  set  of  data  was  not  collected,  the  data  could  not  be 
systematically  tabulated,  and  data  were  not  comparable  across  respondents. 
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First  Revision  of  the  Survey  -  CCIT-II 

Because  of  the  problems  detected  in  the  first  version  of  the  CCIT  survey,  the 
survey  was  revised.  The  second  version  of  the  survey  (CCIT-n)  was  designed  to  collect  a 
more  complete  and  accurate  description  of  incidents  and  produce  more  standardized  data. 
Instead  of  having  respondents  initially  place  their  critical  incidents  into  discrete 
categories  (as  in  the  test  version),  the  CCIT-n  survey  asked  users  to  describe  incidents  in 
an  open-ended  fashion.  This  approach  more  closely  resembles  the  original  critical 
incident  method  described  by  Flanagan  (1954).  It  also  avoids  the  problem  of  different 
respondents  placing  seemingly  similar  incidents  into  different  categories.  In  keeping  with 
Flanagan’s  method,  the  CCIT-H  survey  was  designed  to  focus  respondents’  efforts  on 
describing  the  events  in  detail,  with  the  idea  that  judges  could  be  used  post-hoc  to 
examine  the  data  and  categorize  the  incidents  and  possible  causes.  In  CCIT-II, 
respondents  spent  less  time  categorizing  events  fai  a  highly  structured  menu  scheme,  and 
more  time  describing  the  incidents  with  open-ended  text  boxes.  In  the  CCIT-II  survey, 
most  of  the  questions  were  posed  to  ^  respondents,  thus  ensuring  comparability  of  data 
across  respondents. 

In  the  CCIT-n  survey,  definitions  of  key  terms  were  presented  to  users  in  the 
introductory  portion  of  the  survey.  A  critical  incident  was  defined  as  “an  event  that  did 
cause  or  could  have  caused  an  injury  or  other  negative  outcome  to  a  patient  as  a  result  of 
medical  care.  ’  It  was  indicated  that  “a  critical  incident  can  either  be  an  adverse  event,  in 
which  there  was  an  negative  consequence  for  the  patient,  or  a  near  miss  in  which  there 
could  have  been  a  negative  consequence  but  the  problem  was  detected  and  fixed  in  time.” 
Negative  consequences  experienced  by  patients  could  include  the  following:  discomfort. 
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stress,  increased  hospital  stay,  increased  severity  of  the  illness,  a  need  for  more  treatment, 
development  of  a  new  condition,  disability  or  death.  These  expanded  definitions  of 
“adverse  event”  and  “near  miss”  were  added  to  the  CCIT-n  survey  because  examination 
of  CCIT-I  survey  data  suggested  that  some  respondents  had  failed  to  understand  the 
meaning  of  these  key  terms.  Also,  the  category  called  “positive  events”  was  dropped 
from  the  CCIT  survey  because  only  a  small  number  of  positive  events  were  reported  with 
the  CCIT-I  survey,  and  because  the  events  that  were  reported  did  not  seem  relevant  to  the 
prevention  of  iatrogenic  illness  and  injury. 

In  the  first  version  of  the  CCIT  survey,  there  was  no  way  to  determine  how  many 
different  respondents  had  reported  critical  incidents.  Because  it  is  possible  that  a  small 
number  of  respondents  were  responsible  for  the  bulk  of  the  incident  reports,  it  was 
decided  that  the  revised  CCIT  survey  should  have  a  method  for  determining  how  many 
reports  were  from  the  same  respondent.  Therefore,  we  had  respondents  generate  their 
own  identification  (ID)  numbers,  which  they  were  to  type  in  each  time  they  completed  a 
survey.  The  self-generated  ID  would  be  unknown  to  anyone  other  than  the  respondent.  It 
consisted  of  8  digits:  4  digits  representing  the  respondent’s  mother’s  birthday,  2  digits 
indicating  the  number  of  people  in  the  respondent’s  immediate  family  when  s/he  was  age 
10,  and  2  digits  indicating  the  respondent’s  age  when  s/he  first  owned  a  car  (see 
Appendix  A  for  more  detail).  The  purpose  of  the  self-generated  ID  was  to  allow  incident 
reports  to  be  linked  or  tallied  for  a  single  respondent,  and  to  allow  a  determination  of 
sample  size. 

One  final  change  from  the  first  to  the  second  version  of  the  CCIT  survey  was  to 
convert  it  from  a  Macintosh  to  a  PC  DOS-based  format.  The  DOS  version  of  the  Ci3 
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System  for  Computer  Interviewing  by  Sawtooth  Technologies  (Sawtooth  Software,  1994) 
was  used  to  program  and  administer  the  survey.  Since  DOS-based  systems  are 
commonplace  in  military  and  civilian  hospital  environments,  this  conversion  was 
performed  to  increase  survey  availability  and  allow  greater  expansion  and  modification  of 
the  CCIT  survey  for  use  in  other  institutions. 

Evaluation  of  the  CCIT-II  Survey  by  Subject  Matter  Experts 

The  revised  version  of  the  survey  (CCIT-II)  was  evaluated  by  subject  matter 
experts  (SMEs)  to  obtain  users’  reactions  to  the  survey,  assess  content  validity,  and  detect 
problems.  Eight  emergency  medical  responders  served  as  the  SMEs  (7  males  and  1 
female).  SMEs  were  instructed  to  complete  several  CCIT-n  surveys  each,  using  critical 
incidents  and  workplace  concerns  drawn  from  their  own  professional  experiences.  They 
were  given  a  hand-out  that  listed  the  questions  they  would  be  asked  in  the  post-evaluation 
interview.  They  were  asked  to  make  notes  while  completing  the  survey  of  their 
impressions  and  any  problems  that  they  noticed. 

After  evaluating  the  survey  (which  they  could  do  over  a  one  to  two  week  time 
period),  each  SME  was  interviewed  individually  by  one  of  the  researchers.  The 
interviews  lasted  approximately  30  minutes.  In  the  interviews,  SMEs  were  first  asked 
about  their  overall  impressions  of  the  survey  (positive  or  negative).  Subsequent  questions 
addressed  specific  aspects  of  survey  presentation  and  content.  They  were  asked,  for 
example,  how  easy  or  difficult  they  found  it  to  complete  the  survey  and  their  opinion  of 
the  computer-administered  format.  They  were  asked  if  they  had  suggestions  for  changing 
individual  survey  questions  (i.e.,  could  some  of  the  questions  be  reworded  to  make  them 
easier  to  answer?  How?).  They  were  asked  if  they  thought  the  survey  was  too  long 
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and/or  time-consuming;  if  the  survey  was  comprehensive  enough;  whether  the 
instructions  were  clear;  whether  the  definitions  of  key  terms  (e.g.,  adverse  event)  were 
clear;  whether  the  survey  questions  seemed  redundant;  and  whether  the  medical 
terminology  was  appropriate.  The  SMEs  were  also  asked  if  potential  survey  users 
completing  the  survey  in  their  own  workplace  might  feel  uncomfortable  recording  real 
life  incidents.  Finally,  they  were  asked  if  they  thought  that  use  of  such  a  survey  by  health 
care  facilities  would  improve  risk  management  and  quality  of  care. 

Results  of  Subject  Matter  Experts  Evaluations 

In  general,  the  subject  matter  experts  had  mostly  positive  comments  and  feedback 
about  the  survey.  To  summarize,  six  of  the  eight  SMEs  indicated  without  qualification 
that  they  liked  the  survey  and  all  eight  believed  its  adoption  by  clinical  care  facilities 
could  greatly  improve  risk  management  and  quality  of  care.  Moreover,  all  of  the  SMEs 
liked  the  computer  format  and  thought  it  was  easier,  more  enjoyable  and  more  practical  to 
use  than  a  paper-and-pencil  survey.  While  none  of  the  SMEs  felt  the  survey  was  too 
long,  nearly  all  of  the  experts  thought  the  survey  was  sufficiently  comprehensive. 

Some  of  the  SMEs  recommended  giving  survey  users  more  introductory 
information  on  the  purpose  of  the  survey,  how  their  answers  would  be  evaluated,  to 
whom  the  data  would  be  shown,  and  what  actions  their  organization  would  take  after 
receiving  the  results.  In  addition,  three  of  the  SMEs  expressed  concerns  relating  to 
respondent  anonymity:  they  asked  us  to  give  users  more  assurance  that  their  anonymity 
would  be  protected  and  that  their  data  would  not  be  traced  back  or  linked  to  them.  Two 
of  the  SMEs  believed  that  the  most  sensitive  questions,  or  the  questions  most  likely  to 
jeopardize  anonymity  (“Who  discovered  the  problem?”  and  “What  was  your  role  in  the 
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incident?”)  should  be  made  optional.  Three  of  the  SMEs  believed  that  the  demographic 
items  should  be  made  optional  or  removed;  one  recommended  that  we  explain  the 
purpose  of  the  demographic  items  (e.g.,  for  research  purposes  only).  Two  of  the  SMEs 
recommended  that  we  explain  the  purpose  of  the  self-generated  ID  number  (e.g.,  for 
research  purposes  only). 

The  SMEs  identified  several  ambiguous  phrases  in  the  survey  that  needed  to  be 
clarified  and  made  recommendations  to  improve  the  wording  of  some  survey  questions. 
They  also  identified  multiple-choice  questions  that  needed  more  options  to  cover  all  of 
the  possible  responses.  A  number  of  the  SMEs  noted  that  some  of  the  open-ended  text 
boxes  did  not  allow  enough  space  for  the  amount  of  narrative  they  wanted  to  write. 
Several  of  them  said  they  had  found  it  difficult  to  answer  the  questions  that  used  7-point 
Likert  scales  because  of  the  large  number  of  scale  points  and  also  because  a  “Don’t 
Know”  option  was  not  offered.  Three  of  the  SMEs  identified  questions  that  were  highly 
redundant  and  suggested  that  we  eliminate  some  of  these  questions.  Finally,  three  of  the 
SMEs  stressed  the  potential  value  of  the  Workplace  Concerns  section  and  suggested  it  be 
expanded  to  elicit  more  detailed  information  on  workplace  concerns. 

Second  Revision  of  the  Survey  -  CCIT-III 

Based  on  the  feedback  from  the  SMEs,  the  CCIT  survey  was  revised.  Because  the 
SMEs  did  not  have  a  large  number  of  major  criticisms,  the  changes  made  from  the  second 
to  the  third  version  of  the  survey  were  relatively  minor.  Sawtooth  Software’s  Ci3  System 
for  Computer  Interviewing  was  retained  as  the  software  for  design  and  administration  of 
the  survey.  The  final  version  of  the  survey,  which  incorporates  these  changes,  is 
presented  in  Appendix  A. 
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The  following  changes  were  made  to  the  CCIT  survey.  In  the  introductory  portion 
of  the  survey,  statements  were  added  to  assure  respondents  that  their  anonymity  would  be 
preserved  and  that  no  one  in  the  health  care  facility  would  have  access  to  their  individual 
data  or  would  be  able  to  trace  their  data  to  them.  Some  of  the  more  specific  demographic 
items  were  eliminated  and  respondents  were  given  an  option  to  skip  all  of  the 
demographic  questions.  A  statement  about  the  purpose  of  the  demographic  questions  was 
added:  “This  information  is  for  statistical  purposes  only.  It  will  not  be  used  to  identify 
you”. 

A  number  of  other  changes  were  also  made  to  the  survey.  An  explanation  of  the 
need  for  the  self-generated  ID  number  was  added.  The  Likert  scales  were  changed  from 
7-point  to  4-point  scales  to  make  it  easier  for  respondents  to  answer,  and  a  “Don’t  Know” 
option  was  added.  Some  of  the  questions  identified  as  redundant  by  the  SMEs  were 
eliminated  from  the  survey.  For  some  questions  (those  noted  by  the  SMEs  to  have 
insufficient  space),  additional  space  was  added  to  the  open-ended  text  boxes.  Several 
questions  were  reworded  in  accordance  with  the  SMEs  recommendations  to  improve  item 
clarity.  Instructions  were  also  added  on  how  to  report  an  incident  involving  a  mistake 
that  did  not  lead  to  a  negative  outcome  for  a  patient.  Finally,  the  Workplace  Concerns 
section  of  the  survey  was  expanded  to  allow  more  detailed  information  about  the  concern 
to  be  gathered. 

DISCUSSION 

It  is  widely  recognized  that  iatrogenic  illness  and  injury  are  a  major  source  of 
morbidity  and  mortality  for  hospitalized  patients.  The  first  step  towards  preventing 
adverse  medical  events  is  to  determine  the  types  of  adverse  events  that  occur  most  often 
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and  the  factors  that  lead  up  to  them.  Detailed  knowledge  of  adverse  events  that  occur  in 
real-life  clinical  settings  and  their  underlying  causes  may  lead  to  changes  that  could 
substantially  improve  patient  care. 

In  the  present  project,  the  CCIT  survey  was  developed  to  systematically  collect 
information  about  adverse  medical  events.  The  survey  was  developed  using  Flanagan’s 
(1954)  critical  incident  technique,  and  was  computer-administered.  The  test  version  of 
the  CCrr  survey  (CCIT-I)  collected  incident  reports  over  a  26-month  period  in  the  ICU  of 
a  large  military  hospital.  The  survey  was  positively  received  by  staff  and  the  incident 
reports  obtained  from  the  survey  led  to  a  number  of  corrective  actions  being  taken. 
However,  a  careful  examination  of  the  CCIT-I  survey  revealed  several  significant 
problems.  These  problems  included  classification  problems,  lack  of  comparability  in  data 
collected  across  respondents,  and  closed-ended  questions  that  did  not  elicit  enough  detail 
about  adverse  events.  Therefore,  the  survey  was  revised  to  correct  these  deficiencies. 

The  second  version  of  the  CCIT  survey  (CCTT-n)  was  evaluated  by  subject  matter  experts 
(SMEs),  who  gave  comments  and  recommendations  about  how  the  survey  could  be 
improved.  These  comments  and  recommendations  were  used  to  revise  the  survey  again, 
resulting  in  an  improved,  “final”  version  of  the  CCIT  survey  (CCIT-in),  which  appears  in 
Appendix  A. 

The  subject  matter  experts  who  evaluated  the  CCIT  survey  had  mostly  positive 
comments  about  it.  They  liked  the  computer-administered  format,  did  not  think  the 
survey  was  too  long,  and  thought  it  was  comprehensive.  The  SMEs  had  a  number  of 
suggestions  and  recommendations  for  improving  the  survey  and  for  clarifying  some 
specific  questions.  Interestingly,  all  of  the  SMEs  believed  that  it  would  be  worthwhile  for 
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the  CCIT  survey  or  a  similar  survey  to  be  implemented  by  health  care  facilities  to 
improve  risk  management  and  patient  care. 

Several  of  the  SMEs  suggested  that  when  this  survey  is  implemented  in  a  work 
environment,  the  following  questions  should  be  answered  for  potential  respondents:  (1) 
What  is  the  purpose  of  the  survey  (research,  risk  management,  etc.)?;  (2)  What  will  be 
done  with  the  information  obtained  via  the  survey— will  it  be  used  to  make  changes?;  (3) 
What  kind  of  feedback,  if  any,  will  users  receive  after  the  results  are  compiled?;  (4)  How 
will  the  data  be  made  available  to  the  organization— i.e.,  will  the  hospital  ever  have  access 
to  the  raw  data  or  only  summary  results?;  (5)  Can  anonymity  of  responses  be  absolutely 
guaranteed?  and  (6)  Does  this  survey  replace  the  existing  incident  reporting  system?  The 
SMEs  were  enthusiastic  about  the  survey,  yet  they  believed  that  these  points  should  be 
fully  addressed. 

As  noted  above,  a  number  of  the  SMEs  expressed  concerns  about  respondent 
anonymity.  They  strongly  believed  that  some  medical  care  providers  might  be  hesitant  to 
describe  incidents  that  could  establish  culpability.  Because  anonymity  is  so  important, 
the  most  recent  version  of  the  CCIT  survey  asks  only  a  few  demographic  questions  and 
allows  respondents  to  skip  some  or  even  all  of  these  questions.  For  the  same  reason,  the 
survey  does  not  require  respondents  to  give  detailed  information  (e.g.,  date,  time)  on  the 
incident.  Feedback  from  the  SMEs  clearly  demonstrated  that  if  accurate  data  are  to  be 
gathered,  maintaining  respondents’  anonymity  is  of  paramount  importance  in  the  CCIT. 

The  CCIT  survey  differs  from  existing  quality  assurance  systems  that  hospitals 
have  because:  (1)  it  is  completely  anonymous;  (2)  it  requires  no  paperwork  and  (3)  it  is 
primarily  a  research  tool.  In  contrast,  hospital  quality  assurance  systems  may  or  may  not 
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be  anonymous,  require  a  substantial  amount  of  paperwork,  and  are  not  conducted  for 
research  purposes.  If  implemented  in  hospitals  or  other  health  care  settings,  the  CCIT 
survey  would  augment  but  not  replace  existing  quality  assurance  systems. 

An  important  strength  of  the  CCIT  survey  is  its  inclusion  of  near  misses.  Most 
research  on  medical  mishaps  has  focused  on  adverse  events  only;  near  misses  have  rarely 
been  studied.  Based  on  research  from  aviation  and  other  industries,  there  seems  to  be 
considerable  value  in  the  study  of  near  misses  (Perrow,  1984).  Inclusion  of  near  misses 
allows  information  on  a  larger  and  broader  set  of  incidents  to  be  gathered  and  allows  root 
causes  to  be  detected  fairly  early  in  the  chain  of  accident  evolution. 

Another  strength  of  the  CCIT  survey  is  its  assessment  of  workplace  concerns, 
along  with  near  misses  and  adverse  events.  In  the  present  study,  a  workplace  concern  was 
defined  as  “any  issue  or  concern  regarding  the  workplace  environment.”  Most  of  the 
workplace  concerns  expressed  by  survey  respondents  were  concerns  about  other  workers 
not  doing  their  jobs  properly,  problems  with  supervisors,  supply  problems,  environmental 
problems  (e.g.,  poor  lighting),  and  problems  with  hospital  procedures.  Because  each  of 
these  factors  could  potentially  play  a  role  in  activating  or  sustaining  the  chain  of  events 
leading  up  to  an  adverse  event,  it  is  important  that  they  be  identified,  studied,  and 
understood. 

The  potential  for  obtaining  valuable  information  about  adverse  medical  events 
using  a  survey  tool  such  as  the  CCIT  survey  appears  promising.  Based  on  our  experience, 
a  much  greater  quality  and  quantity  of  incident  data  can  be  expected  using  this  approach 
than  could  be  obtained  via  review  of  medical  charts  or  legally  required  hospital  incident 
report  forms.  This  conclusion  was  also  expressed  by  some  of  the  SMEs  who  evaluated 
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the  CCIT  survey.  We  recommend  that  the  CCIT  or  a  similar  survey  be  used  in  clinical 
care  settings.  Such  a  survey  could  be  implemented  in  the  Department  of  Defense’s 
existing  bedside  clinical  integrated  work  stations.  Integration  into  existing  systems  would 
make  it  easy  for  health  care  professionals  to  record  near  misses  and  adverse  events  as  they 
occur. 

For  health  care  facilities  to  successfully  implement  this  survey  approach,  several 
thorny  issues  will  have  to  be  addressed.  First,  users  of  the  CCIT  survey  must  be  prepared 
to  have  one  or  more  experts  interpret  and  follow  up  on  critical  incidents  assessed  by  the 
survey;  the  CCIT  survey  will  not  automatically  summarize  the  data.  In  addition, 
implementation  of  a  survey  like  the  CCIT  will  cause  facility  administrators  to  have  to 
grapple  with  questions  of  anonymity  and  respondent  hability,  medico-legal  implications, 
and  workplace  morale.  Finally,  a  commitment  from  all  levels  of  management  regarding 
the  importance  of  the  survey  is  necessary  to  ensure  adequate  participation  of  health  care 
personnel  and  to  protect  these  personnel  from  negative  repercussions  as  a  result  of 
disclosing  adverse  events. 
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Appendix  A 

Computerized  Critical  Incident  Technique  Survey 


A 


1 


WELCOME  TO  THE  COMPUTERIZED  CRITICAL 
INCIDENT  TECHNIQUE  (CCIT)  SURVEY 


This  survey  system  has  two  purposes: 

It  allows  you  to  report  "critical  incidents"  involving  hospitalized  patients. 

It  allows  you  to  report  workplace  concerns. 

TO  EXIT  COMPLETELY,  press  Ctrl-End.  If  you  want  to  exit  at  some  point  later  in  the 
survey,  try  pressing  Ctrl-End  twice. 

Press  any  key  to  continue. 


2 

You  may  choose  to  use  the  system  for  ONE  or  for  BOTH  of  these  purposes  (critical 
incidents  or  workplace  concerns). 

Both  reports  go  into  a  centralized  data  base  which  will  be  used  to  detect  problems  and 
make  positive  changes  in  health  care  and  in  the  working  environment. 

Press  any  key  to  continue. 


3 

A  CRITICAL  INCIDENT  is  an  event  that  did  cause  or  could  have  caused  an  injury  or 
other  negative  outcome  to  a  patient  as  a  result  of  medical  care.  Such  an  incident  involves 
actual  or  potential  harm  to  a  patient  as  a  result  of  medical  treatment  or  care  s/he  received 
or  failed  to  receive. 

Another  term  for  a  critical  incident  occurring  in  a  medical  setting  is  medical  mishap. 

Press  any  key  to  continue. 
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A  critical  incident  can  either  be  an  ADVERSE  EVENT,  in  which  there  was  a  negative 
consequence  for  the  patient  or  a  NEAR  MISS,  in  which  there  could  have  been  a  negative 
consequence  but  the  problem  was  detected  and  fixed  in  time. 

Negative  consequences  experienced  by  patients  might  include  discomfort  or  stress  for  the 
patient,  increased  hospital  stay,  increased  severity  of  illness,  a  need  for  additional 
treatment,  development  of  a  new  condition,  disability,  or  death. 

Press  any  key  to  continue. 

_ 

The  KEY  FOCUS  of  this  project  is  critical  incidents  occurring  in  intensive  care  units 
(ICUs).  However,  for  this  preliminary  version  of  the  survey,  we  will  accept  incidents  that 
happened  in  intensive  care,  critical  care,  operating  rooms,  or  emergency  rooms. 

Press  any  key  to  continue. 

6  ~~ 

Before  reporting  a  Critical  Incident  ask  yourself: 

Did  I  personally  participate  in  OR  observe  the  incident  first-hand? 

Do  I  remember  the  incident  well  enough  to  describe  it  in  detail? 

If  you  answered  "YES"  to  these  questions,  we  want  you  to  report  the  incident. 

If  not,  you  may  think  of  another  incident  that  meets  these  criteria  or  report  a  workplace 
concern. 

Press  any  key  to  continue. 

7  ^ 

The  OTHER  FOCUS  of  this  survey  system  is  workplace  concerns. 

A  WORKPLACE  CONCERN  is  any  other  issue  or  concern  regarding  people  in  the 
workplace  or  the  workplace  environment. 

A  WORKPLACE  CONCERN  is  generally  an  ongoing  problem  in  the  workplace.  It  might 
have  to  do  with  policies  or  procedures,  working  conditions,  communication,  equipment, 
supplies,  inadequate  functioning  of  a  hospital  service,  or  problem  staff  members. 
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Press  any  key  to  continue. 


8 

If  you  have  any  workplace  concerns  occurring  in  either  intensive  care,  critical  care,  the 
operating  room  or  emergency,  you  can  report  them  using  this  system. 

Press  any  key  to  continue. 


9 

Be  assured  that  no  one  in  this  or  any  other  health  care  facihty  will  have  access  to  the 
information  that  you  enter  into  this  survey  system.  Only  summary  results  (grouped  across 
survey  respondents)  will  ever  be  reported.  No  one  in  your  organization  will  ever  be  able 
to  link  the  information  that  you  report  back  to  you  as  an  individual. 

Press  any  key  to  continue. 


10 

Do  you  want  to  enter  either  a  critical  incident  or  a  workplace 
concern? 

Press  Y  for  YES  to  continue  or  N  for  No  to  Exit  the  system. 
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In  order  to  link  multiple  workplace  concerns  and  critical  incident  reports  from  the  same 
person,  you  are  asked  to  generate  an  ID  number. 

This  ID  cannot  be  used  to  identify  you— it  is  for  research  purposes  only.  Its  purpose  is  to 
link  your  data  if  you  complete  more  than  one  report  using  this  system  (e.g.,  to  tally  up 
number  of  reports  and  number  of  reports  per  respondent). 

Press  any  key  to  continue. 


12 

The  ID  will  consist  of  the  following,  in  the  order  shown: 

—a  4-digit  number  indicating  your  mother's  birthday  (2  digits  for  MONTH,  2  digits  for 
DAY).  March  20  would  be  "0320". 
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-a  2-digit  number  indicating  the  number  of  people  who  were  in  your  immediate  family 
(including  you)  when  you  were  age  10.  If  your  family  was  made  up  of  you  plus  3  other 
members,  you  would  type  "04"  (be  sure  to  include  the  zero). 

-your  age  when  you  first  owned  a  car.  Type  00  if  you  have  never  owned  one.  If  you  first 
owned  a  car  at  age  19,  you  would  type  "  19". 

In  the  above  example,  the  ID  would  be  03200419. 

Type  these  8  characters  in  the  space  below  and  press  ENTER. 

If  you  make  a  mistake,  use  the  BACKSPACE  key  and  retype. 


13 

If  you  have  already  completed  this  survey  at  least  once,  you  may  wish  to  skip  the  next  few 
introductory  screens. 

Do  you  want  to  go  directly  to  the  screen  that  lets  you  choose  a  critical  incident  versus  a 
workplace  concern? 

Press  Y  for  YES  or  N  for  No. 

14  ^ 

You  may  use  the  system  as  often  as  you  wish.  However,  please  do  not  report  the  same 
CRITICAL  INCIDENT  more  than  once  unless  the  incident  happened  more  than  once. 

Press  any  key  to  continue. 

—  ^  —  __ 

You  will  be  answering  questions  by  following  the  instructions  that  appear  on  each  screen. 

If  you  want  to  go  back  and  review  an  answer  to  a  previous  question  or  look  at  a  previous 
screen,  press  the  "ESC"  key  (usually  in  the  upper  left-hand  side  of  the  keyboard).  ESC 
allows  you  to  backtrack  within  the  survey.  To  retain  a  previous  answer  after 
backtracking,  simply  press  ENTER-this  will  move  you  forward  in  the  survey.  To  change 
a  previous  answer,  type  in  the  new  answer  and  it  will  replace  your  old  one. 

Press  any  key  to  continue. 
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When  typing  answers  to  the  survey  questions,  mixing  upper-  and  lower-case  letters  is 
okay. 

You  can  move  around  the  text  you've  typed  and  edit  by  using  the  arrow  keys. 

Press  any  key  to  continue. 


17 

If  you  wish  to  make  both  types  of  reports  (critical  incident  and  workplace  concern), 
simply  pick  EITHER  critical  incident  or  workplace  concern  from  the  menu  that  follows 
and  complete  the  report. 

When  you  have  completed  the  report,  you  will  be  returned  to  the  main  menu.  At  that 
point,  simply  pick  the  OTHER  type  of  report. 

Press  any  key  to  continue. 


18 

What  type  of  report  do  you  wish  to  make? 

1.  Critical  Incident 

2.  Workplace  Concern 

3.  None-Please  exit  me  from  the  system 

Press  1,  2,  or  3. 


CRITICAL  INCIDENT  SECTION 


19 

A  CRITICAL  INCIDENT  is  a  specific  event  involving  a  patient  that  had  or  could  have 
had  a  negative  impact  on  the  patient. 

Some  critical  incidents  are  due  to  poor  patient  care  but  many  occur  despite  good  care 
(e.g.,  an  unexpected  drug  reaction). 


Press  any  key  to  continue. 
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A  critical  incident  can  be  the  result  of  something  that  was  DONE  to  a  patient  or  due  to 
something  that  was  NOT  DONE  (e.g.,  failure  to  monitor  a  patient). 

Examples  of  critical  incidents  include  medication  errors,  problems  with  equipment  or 
supplies,  communication  problems,  problems  with  medical  procedures,  problems  with 
hospital  services  (e.g.,  lab),  and  problems  with  staff  members’  performance. 

Press  any  key  to  continue. 

—  — - 

If  the  incident  you  have  in  mind  involved  a  mistake  or  deviation  from  accepted  standards 
of  care  but  it  did  not  lead  to  a  negative  outcome  for  a  patient,  you  may  choose  to  report  it 
either  as  a  critical  incident  (near  miss)  or  as  a  workplace  concern. 

Report  it  as  a  critical  incident  if  you  think  it  is  likely  that  this  type  of  mistake  could  ever  be 
clinically  significant  and  lead  to  a  negative  outcome  for  a  patient.  Report  it  as  a  workplace 
concern  if  you  think  it  is  unlikely  that  this  t5q)e  of  mistake  could  ever  lead  to  a  negative 
outcome  for  a  patient. 

Press  1  to  go  to  the  section  on  Workplace  Concerns. 

Press  2  to  continue  the  Critical  Incident  section  of  the  survey. 

22  ■ 

We  want  you  to  report  the  most  recent  critical  incidents  you  can  recall. 

If  you  are  trying  to  decide  which  of  2  or  3  different  incidents  to  report,  please  report  the 
one  that  occurred  MOST  RECENTLY. 

You  are  welcome  to  report  all  incidents  that  meet  the  criteria:  (1)  you  personally 
participated  or  observed  it,  and  (2)  you  remember  it  well  enough  to  describe  it  in  detail. 

Do  you  have  an  incident  in  mind  that  you  want  to  report? 

Press  Y  for  YES  or  N  for  No. 


23 

When  answering  the  questions  that  follow,  please  define  each  abbreviation  you  use  so  that 
your  answers  will  be  clear  to  NONMEDICAL  personnel. 
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When  answering  any  open-ended  question  (essay-type),  you  can  always  type  “Not 
Applicable”  when  appropriate.  Also,  if  you  have  already  answered  the  question  being 
posed,  please  indicate  this  by  typing  “See  Previous”  or  something  similar.  You  do  NOT 
need  to  specify  the  previous  screen/question. 

Thanks  for  your  patience! 

Press  any  key  to  continue. 


24 

What  happened?  Explain  what  happened  from  YOUR  OWN  PERSPECTIVE  (e.g.,’T 
noticed  that  the  patient  was  not  given  medication  on  time" . . .). 

Explain  WHO  did  WHAT  using  people’s  positions  to  refer  to  them  rather  than  their  names 
(e.g.,  "An  MD  made  a  ventilator  change  . . ."). 

Type  your  answer  in  the  space  below.  When  you  reach  the  end  of  the  line,  keep  typing 
(you  don’t  need  to  press  enter).  When  finished,  press  ENTER  twice. 


25 

Who  discovered  the  problem?  (e.g.,  a  nurse,  a  physician,  etc.) 

You  can  skip  this  question  if  you  choose.  Do  you  want  to  answer  this  question? 
Press  Y  for  YES  to  continue  or  N  for  No  to  skip  this  question. 


26 

Who  discovered  the  problem?  (e.g.,  a  nurse,  a  physician,  etc.) 

Type  your  answer  in  the  space  below.  When  finished,  press  ENTER  twice. 
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What  was  your  role  or  position  with  regard  to  this  incident/patient  (e.g.,  "I  was  the 
patient's  bedside  nurse")? 

You  can  skip  this  question  if  you  choose.  Do  you  want  to  answer  this  question? 
Press  Y  for  YES  to  continue  or  N  for  No  to  skip  this  question. 

28  ■ 

What  was  your  role  or  position  with  regard  to  this  incident/patient  (e.g.,  "I  was  the 
patient's  bedside  nurse")? 

Type  your  answer  in  the  space  below.  When  finished,  press  ENTER  twice. 


29 

How  many  days  ago  (approximately)  did  the  incident  occur? 
Type  the  number  in  the  space  below  and  press  ENTER. 


30 

Do  you  think  this  event  could  have  been  avoided  or  prevented? 

1.  Definitely 

2.  Probably 

3.  Probably  not 

4.  Definitely  not 

5.  Don't  know 

Press  a  number  between  1  and  5. 

31  - - 

Explain  why  you  think  this  event  possibly  could  have  been  AVODDED  or  PREVENTED. 
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Type  your  answer  in  the  space  below.  When  finished,  press  ENTER  twice. 


32 

Explain  why  you  think  this  event  probably  could  NOT  have  been  AVOIDED  or 
PREVENTED. 

Type  your  answer  in  the  space  below.  When  finished,  press  ENTER  twice. 


33 

Please  explain  your  answer  if  necessary— i.e.,  why  aren't  you  sure  if  this  event  could  have 
been  AVOIDED  or  PREVENTED? 

Type  your  answer  in  the  space  below.  When  finished,  press  ENTER  twice. 


34 

What  do  you  think  was  the  most  important  cause  of  this  incident? 

Type  your  answer  in  the  space  below.  When  finished,  press  ENTER  twice. 
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Describe  any  other  factors  which  may  have  played  a  role  in  the  incident. 

If  you  cannot  think  of  any  others,  type  "None". 

Type  your  answer  in  the  space  below.  When  finished,  press  ENTER  twice. 


36 

What  was  the  sex  of  the  patient? 

1. Male 

2.  Female 

3.  Don't  Know 

Press  1,  2  or  3. 

— 

What  was  the  approximate  age  of  the  patient? 

Type  the  number  in  the  space  below  and  press  ENTER. 


38 

What  was  the  patient's  condition  before  the  incident? 

1.  Stable 

2.  Unstable 

3.  Stable  but  potentially  unstable  if  not  treated  promptly 

4.  Comfort  care 

5.  Don't  know 

6.  Other 

Press  a  number  between  1  and  6. 
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What  was/were  the  patient's  main  diagnosis/diagnoses  before  the  critical  incident? 
Type  your  answer  in  the  space  below.  When  finished,  press  ENTER  twice. 


40 

What  were  the  NEGATIVE  CONSEQUENCES  of  the  incident  for  the  patient?  Some 
possible  negative  consequences  include  discomfort,  increased  morbidity,  prolonged 
hospital  or  ICU  stay,  extra  procedures  required,  disability,  or  death. 

If  there  were  no  negative  consequences,  type  "None". 

Type  your  answer  in  the  space  below.  When  finished,  press  ENTER  twice. 


41 

Would  you  say  that  the  incident  was  an  ADVERSE  EVENT  or  a  NEAR  MISS? 

In  an  ADVERSE  EVENT,  there  is  some  negative  consequence  for  the  patient  (which  can 
include  discomfort). 

A  NEAR  MISS  is  a  situation  that  could  have  led  to  a  negative  consequence  for  the  patient 
but  did  not  (either  due  to  chance  or  because  the  problem  or  error  was  detected  and  fixed 
in  time). 

I  would  classify  the  incident  as  a(n): 

1.  Adverse  Event 

2.  Near  Miss 

3.  Other  or  Neither 

Press  1, 2,  or  3. 
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Do  you  think  that  the  incident  was  life  threatening? 

1.  Definitely 

2.  Probably 

3.  Probably  not 

4.  Definitely  not 

5.  Don't  know 

Press  a  number  between  1  and  5. 

43  “ 

Please  give  the  positions  of  the  personnel  who  were  directly  involved  in  the  incident  (e.g., 
1  staff  physician,  1  nurse,  and  1  intern).  Be  as  specific  as  possible.  (Include  YOURSELF 
if  you  were  involved.) 

Press  ENTER  twice  when  done. 


44 

How  involved  were  you  in  the  incident? 

1 . 1  was  directly  involved 

2. 1  was  NOT  involved  but  I  OBSERVED  the  incident 

3. 1  was  NOT  involved,  NOR  did  I  observe  the  incident  (I  heard  about  it  from 
someone  else) 

4.  Other 

Press  a  number  between  1  and  4. 

45  ~ 

Do  you  think  this  event  involved  human  error? 


1.  Definitely 

2.  Probably 

3.  Probably  not 
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4.  Definitely  not 

5.  Don't  know 

Press  a  number  between  1  and  5. 


46 

Explain  why  you  think  human  error  WAS  or  WAS  NOT  involved.  (Type  “see  previous”  if 
you  already  gave  this  information.) 

Type  your  answer  in  the  space  below.  When  finished,  press  ENTER  twice. 
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Do  you  think  the  incident  involved  ineffective  performance  by  a  staff  member? 

1.  Definitely 

2.  Probably 

3.  Probably  not 

4.  Definitely  not 

5.  Don't  know 

Press  a  number  between  1  and  5. 
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Did  the  incident  involve  a  problem  with  MEDICATION  (such  as  a  medication  error  or  a 
failure  to  medicate)  or  BLOOD/BLOOD  PRODUCTS? 

Press  Y  for  YES  or  N  for  No. 

__ 

What  specific  MEDICATION(S)  or  BLOOD/BLOOD  PRODUCTS  were  involved? 
Type  your  answer  in  the  space  below.  When  finished,  press  ENTER  twice. 
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48.2 

If  a  medication,  what  general  category  of  MEDICATION(S)  was  involved?  (such  as 
antibiotics).  If  a  medication  was  not  involved,  type  "NA"  for  "Not  Applicable". 

Type  your  answer  in  the  space  below.  When  finished,  press  ENTER  twice. 


48.3 

Do  you  have  any  other  comments  you  wish  to  make  about  the  MEDICATION(S)  or 
BLOOD/BLOOD  PRODUCTS  involved  in  this  incident? 

Press  Y  for  YES  or  N  for  No. 

484  ^  - - 

Type  your  additional  comments  in  the  space  below.  When  finished,  press  ENTER  twice. 


49 

Did  the  incident  involve  a  problem  with  a  PROCEDURE? 

Press  Y  for  YES  or  N  for  No. 

49J  ~ 

Explain  how  this  PROCEDURE  caused  a  problem. 

Type  your  answer  in  the  space  below.  When  finished,  press  ENTER  twice. 
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49.2 

What  PROCEDURE  was  involved? 

Type  your  answer  in  the  space  below.  When  finished,  press  ENTER  twice. 


49.3 

Was  the  PROCEDURE  appropriate? 

1.  Definitely 

2.  Probably 

3.  Probably  not 

4.  Definitely  not 

5.  Don't  know 

6.  Not  applicable 

Press  a  number  between  1  and  6. 


49.4 

Was  the  PROCEDURE  done  competently? 

1.  Definitely 

2.  Probably 

3.  Probably  not 

4.  Definitely  not 

5.  Don't  know 

6.  Not  applicable 

Press  a  number  between  1  and  6. 


49.5 

Should  a  PROCEDURE  have  been  done  that  was  not  done? 
Press  Y  for  YES  or  N  for  No. 
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49.6 


Do  you  have  any  other  comments  you  wish  to  make  about  the  PROCEDURE(S)  involved 
in  this  incident? 

Press  Y  for  YES  or  N  for  No. 

49.7  ^ 

Type  your  additional  comments  in  the  space  below.  When  finished,  press  ENTER  twice. 


50 

Did  the  incident  involve  a  problem  with  EQUIPMENT  or  SUPPLIES  (or  lack  of 
equipment  or  supplies)? 

Press  Y  for  YES  or  N  for  No. 

50J 

What  specific  EQUIPMENT  or  SUPPLIES  were  involved? 

Type  your  answer  in  the  space  below.  When  finished,  press  ENTER  twice. 


50.2 

Do  you  have  any  other  comments  you  wish  to  make  about  the  EQUIPMENT  or 
SUPPLIES  involved  in  this  incident? 

Press  Y  for  YES  or  N  for  No. 

503  ■  ~ 

Type  your  additional  comments  in  the  space  below.  When  finished,  press  ENTER  twice. 
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Did  the  incident  involve  inadequate  functioning  of  a  HOSPITAL  SERVICE  (e.g.,  lab, 
pharmacy,  etc.)? 

Press  Y  for  YES  or  N  for  No. 


51.1 

What  HOSPITAL  SERVICE  was  involved? 

Type  your  answer  in  the  space  below.  When  finished,  press  ENTER  twice. 


51.2 

Do  you  have  any  other  comments  you  wish  to  make  about  the  HOSPITAL  SERVICE 
involved  in  this  incident? 

Press  Y  for  YES  or  N  for  No. 


51.3 

Type  your  additional  comments  in  the  space  below.  When  finished,  press  ENTER  twice. 
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Rate  the  importance  of  the  factor  shown  below  in  causing  the  incident.  Type  in  a  number 
between  1  and  5  to  show  how  IMPORTANT  or  UNIMPORTANT  you  think  the  factor 
was.  You  can  type  in  a  number  that  is  between  rating  points,  such  as  3.5. 

Type  a  zero  (0)  if  you  are  UNSURE  or  DON’T  KNOW. 

After  you  have  typed  in  a  number,  press  ENTER. 

Inadequate  supervision  of  medical  staff 

0  1 - 2 - 3 - 4 - 5 

Don’t  Not  at  all  Moderately  Extremely 

Know  Important  Important  Important 

53 

Rate  the  importance  of  the  factor  shown  below  in  causing  the  incident.  Type  in  a  number 
between  1  and  5  to  show  how  IMPORTANT  or  UNIMPORTANT  you  think  the  factor 
was.  You  can  type  in  a  number  that  is  between  rating  points,  such  as  3.5. 

Type  a  zero  (0)  if  you  are  UNSURE  or  DON’T  KNOW. 

After  you  have  typed  in  a  number,  press  ENTER. 

Poor  morale  of  medical  staff 

0  1 - 2 - 3 - 4 - 5 

Don’t  Not  at  all  Moderately  Extremely 

Know  Important  Important  Important 


54 

Rate  the  importance  of  the  factor  shown  below  in  causing  the  incident.  Type  in  a  number 
between  1  and  7  to  show  how  IMPORTANT  or  UNIMPORTANT  you  think  the  factor 
was.  You  can  type  in  a  number  that  is  between  rating  points,  such  as  3.5. 

Type  a  zero  (0)  if  you  are  UNSURE  or  DON’T  KNOW. 

After  you  have  typed  in  a  number,  press  ENTER. 

Lack  of  teamwork  among  medical  staff 

0  1 - - - 2 - 3 - 4 - 5 

Don’t  Not  at  all  Moderately  Extremely 

Know  Important  Important  Important 


55 

Rate  the  importance  of  the  factor  shown  below  in  causing  the  incident.  Type  in  a  number 
between  1  and  7  to  show  how  IMPORTANT  or  UNIMPORTANT  you  think  the  factor 
was.  You  can  type  in  a  number  that  is  between  rating  points,  such  as  3.5. 

Type  a  zero  (0)  if  you  are  UNSURE  or  DON’T  KNOW. 

After  you  have  typed  in  a  number,  press  ENTER. 

Shortage  of  equipment  or  supplies 

0  1 - 2 - 3 - 4 - 5 

Don’t  Not  at  all  Moderately  Extremely 

Know  Important  Important  Important 
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Rate  the  importance  of  the  factor  shown  below  in  causing  the  incident.  Type  in  a  number 
between  1  and  7  to  show  how  IMPORTANT  or  UNIMPORTANT  you  think  the  factor 
was.  You  can  type  in  a  number  that  is  between  rating  points,  such  as  3.5. 

Type  a  zero  (0)  if  you  are  UNSURE  or  DON’T  KNOW. 

After  you  have  typed  in  a  number,  press  ENTER. 

Faulty  equipment  or  supplies 

0  1 - 2 - 3 - 4 - 5 

Don’t  Not  at  all  Moderately  Extremely 

Know  Important  Important  Important 

57  '  ■ 

Rate  the  importance  of  the  factor  shown  below  in  causing  the  incident.  Type  in  a  number 
between  1  and  7  to  show  how  IMPORTANT  or  UNIMPORTANT  you  think  the  factor 
was.  You  can  type  in  a  number  that  is  between  rating  points,  such  as  3.5. 

Type  a  zero  (0)  if  you  are  UNSURE  or  DON’T  KNOW. 

After  you  have  typed  in  a  number,  press  ENTER. 

Unit  busy  and/or  understaffed 

0  1 - 2 - 3 - 4 - 5 

Don’t  Not  at  all  Moderately  Extremely 

Know  Important  Important  Important 
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58 

Rate  the  importance  of  the  factor  shown  below  in  causing  the  incident.  Type  in  a  number 
between  1  and  7  to  show  how  IMPORTANT  or  UNIMPORTANT  you  think  the  factor 
was.  You  can  type  in  a  number  that  is  between  rating  points,  such  as  3.5. 

Type  a  zero  (0)  if  you  are  UNSURE  or  DON’T  KNOW. 

After  you  have  typed  in  a  number,  press  ENTER. 

Patient’s  behavior  or  actions 

0  1 - 2 - 3 - 4 - 5 

Don’t  Not  at  all  Moderately  Extremely 

Know  Important  Important  Important 


59 

Rate  the  importance  of  the  factor  shown  below  in  causing  the  incident.  Type  in  a  number 
between  1  and  7  to  show  how  IMPORTANT  or  UNIMPORTANT  you  think  the  factor 
was.  You  can  type  in  a  number  that  is  between  rating  points,  such  as  3.5. 

Type  a  zero  (0)  if  you  are  UNSURE  or  DON’T  KNOW. 

After  you  have  typed  in  a  number,  press  ENTER. 

Lack  of  conscientiousness  on  the  part  of  medical  staff 

0  1 - 2 - 3 - 4 - 5 

Don’t  Not  at  all  Moderately  Extremely 

Know  Important  Important  Important 
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Rate  the  importance  of  the  factor  shown  below  in  causing  the  incident.  Type  in  a  number 
between  1  and  7  to  show  how  IMPORTANT  or  UNIMPORTANT  you  think  the  factor 
was.  You  can  type  in  a  number  that  is  between  rating  points,  such  as  3.5. 

Type  a  zero  (0)  if  you  are  UNSURE  or  DON’T  KNOW. 

After  you  have  typed  in  a  number,  press  ENTER. 

Lack  of  skill  on  the  part  of  medical  staff 

0  1 - 2 - 3 - 4 - 5 

Don  t  Not  at  all  Moderately  Extremely 

Know  Important  Important  Important 

—  _ 

Rate  the  importance  of  the  factor  shown  below  in  causing  the  incident.  Type  in  a  number 
between  1  and  7  to  show  how  IMPORTANT  or  UNIMPORTANT  you  think  the  factor 
was.  You  can  type  in  a  number  that  is  between  rating  points,  such  as  3.5. 

Type  a  zero  (0)  if  you  are  UNSURE  or  DON’T  KNOW. 

After  you  have  typed  in  a  number,  press  ENTER. 

Seriousness  of  the  patient's  condition 

0  1 - 2 - 3-— - 4 - 5 

Don  t  Not  at  all  Moderately  Extremely 

Know  Important  Important  Important 


62 

Do  you  have  any  OTHER  suggestions  on  how  events  like  the  one  you  reported  could  be 
prevented  in  the  future? 

If  so,  type  them  below.  Press  ENTER  twice  when  you  are  done. 

If  you  do  not  have  any  suggestions,  simply  press  ENTER  twice. 
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Do  you  have  any  other  comments  about  the  incident? 

If  so,  type  them  below.  Press  ENTER  twice  when  you  are  done. 

If  you  do  not  have  any  other  comments,  simply  press  ENTER  twice. 


WORK  PLACE  CONCERNS  SECTION 


64 

A  WORKPLACE  CONCERN  is  any  issue  or  concern  regarding  the  workplace 
environment. 

A  WORKPLACE  CONCERN  is  generally  an  ongoing  problem  in  the  workplace.  It  might 
have  to  do  with  policies  or  procedures,  working  conditions,  communication,  equipment, 
supplies,  inadequate  functioning  of  a  hospital  service,  or  problem  staff. 

Do  you  have  an  workplace  concern  in  mind  that  you  want  to  report? 

Press  Y  for  YES  or  N  for  No. 


65 

What  was/is  the  problem?  Describe  the  workplace  concern  in  your  own  words. 

Please  describe  ONE  workplace  concern  only.  (You  can  re-enter  the  system  to  describe 
additional  workplace  concerns.) 

Type  your  answer  in  the  space  below.  When  you  reach  the  end  of  the  line,  keep  typing 
(you  don’t  need  to  press  enter).  When  finished,  press  ENTER  twice. 
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Would  you  say  that  this  an  ongoing  problem? 

Press  Y  for  YES  or  N  for  No. 

67  ~ 

How  often  would  you  say  that  this  problem  occurs? 

1.  Almost  all  the  time  (it  is  a  chronic  problem) 

2.  Daily 

3.  Weekly 

4.  Monthly 

5.  Less  often  than  once  a  month 

6.  Don't  know 

Press  a  number  between  1  and  6. 

—  —  ^ 

Where  does/did  this  problem  occur  (e.g.,  ICU,  ER,  etc.)?  If  the  problem  typically  occurs 
in  more  than  one  area,  please  indicate  all  of  them. 

Type  your  answer  in  the  space  below.  When  finished,  press  ENTER  twice. 


69 

What  do  you  think  could  be  done  to  solve  or  minimize  this  problem?  Explain  in  your  own 
words. 

Type  your  answer  in  the  space  below.  When  finished,  press  ENTER  twice. 
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Has  anything  been  done  already  in  an  attempt  to  solve  this  problem?  What? 
Type  your  answer  in  the  space  below.  When  finished,  press  ENTER  twice. 


71 

Is  there  something  that  you  (personally)  could  do  to  improve  this  situation? 
Type  your  answer  in  the  space  below.  When  finished,  press  ENTER  twice. 


72 

In  what  way  or  ways  does  this  concern  adversely  affect  you,  coworkers,  and/or  patients? 
Type  your  answer  in  the  space  below.  When  finished,  press  ENTER  twice. 


73 

Do  you  have  any  other  comments  or  suggestions  you  would  like  to  make  regarding  this 
WORKPLACE  CONCERN? 

If  you  do  not  wish  to  make  any  additional  comments,  simply  press  ENTER  twice. 
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DEMOGRAPHIC  SECTION 


74 

The  following  questions  ask  for  some  demographic  information  about  you.  This 
information  is  for  statistical  purposes  only.  It  will  not  be  used  to  identify  you.  This 
information  is  only  being  gathered  so  that  the  data  can  be  grouped  in  various  ways  when 
statistical  analyses  are  done  (e.g.,  by  sex,  educational  level,  etc.). 

However,  you  can  decline  to  answer  any  single  demographic  question  (e.g.,  your  sex)  by 
choosing  the  “decline  to  answer”  option  presented  after  the  question. 

Press  any  key  to  continue. 

75  “  ^  ~ 

If  you  are  sure  that  you  do  not  want  to  answer  any  of  the  demographic  questions,  you  can 
skip  this  set  of  questions  completely. 

T5q)e  1  to  skip  all  of  the  demographic  questions. 

Type  2  if  you  are  willing  to  consider  answering  some  of  the  demographic  questions. 

76  ^  ^ 

What  is  your  sex? 

1. Male 

2.  Female 

3. 1  decline  to  answer. 

Press  1,  2  or  3. 

77  ^  ~ 

What  is  your  highest  level  of  education? 

1.  Did  not  graduate  from  high  school 

2.  High  school  graduate 

3.  Some  college,  no  degree 

4.  Associate's  degree 

5.  Bachelor's  degree 

6.  Advanced  degree  (beyond  bachelor's) 

7. 1  decline  to  answer. 
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Press  a  number  between  1  and  7. 


78 

What  is  your  position? 

1.  Staff  physician 

2.  Fellow 

3.  Resident 

4.  Intern 

5.  Nurse 

6.  Paramedic 

7.  Corpsman 

8.  Other 

9. 1  decline  to  answer. 

Press  a  number  between  1  and  9. 


79 

How  long  have  your  worked  for  this  organization? 

1.  Less  than  6  months 

2.  Between  6  months  and  18  months 

3.  Between  18  months  and  3  years 

4.  Between  3  and  5  years 

5.  More  than  5  years 
6. 1  decline  to  answer 

Press  a  number  between  1  and  6. 


80 

You  have  completed  the  survey!  Thank  you  very  much  for  your  help. 
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Do  you  have  any  comments  about  this  SURVEY  SYSTEM?  If  so,  type  them  in  the  space 
below.  Press  ENTER  twice  when  you  are  done. 

If  you  do  not  wish  to  make  any  additional  comments,  simply  press  ENTER  twice. 


82 


THANK  YOU  VERY  MUCH  FOR  PARTICIPATING! 


To  exit,  wait  until  this  screen  is  replaced  by  the  Welcome  screen  and  press  Ctrl-End. 
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